A cluster file system allows multiple client devices to share access to files over a network. One well-known cluster file system is the Lustre file system. Lustre is a Linux-based high performance cluster file system utilized for computer clusters ranging in size from small workgroup clusters to large-scale, multi-site clusters. Lustre can readily scale to support tens of thousands of clients, petabytes of storage capacity, and hundreds of gigabytes per second of aggregate input-output (IO) throughput. Due to its high performance and scalability, Lustre is utilized in many supercomputers, as well as other complex computing environments, including large enterprise data centers.
In conventional Lustre implementations, it can be difficult to balance the conflicting requirements of storage capacity and IO throughput. IO operations on object storage servers are generally performed directly with back-end storage arrays associated with those servers, and the corresponding storage devices may not be well matched to the current needs of the system. This can lead to situations in which either performance is less than optimal or the costs of implementing the system become excessive.
Traditional file systems typically store data on disks and then retrieve the stored data from the disks. Virtual file systems, however, do not actually store data. Rather, virtual file systems act as a view or translation of an existing file system or storage device. FUSE (File System in Userspace) is particularly useful for writing virtual file systems. FUSE is an existing file system abstraction with a well-defined interface. In principle, any resource that is available to a FUSE implementation can be exported as a file system. Parallel Log Structured File System (PLFS) and many other file systems have FUSE implementations.
Accordingly, despite the many advantages of Lustre file systems and other similar cluster file systems, a need remains for additional improvements, particularly with regard to IO operations. For example, further acceleration of IO operations, leading to enhanced system performance relative to conventional arrangements, would be desirable. Additionally or alternatively, an ability to achieve particular levels of performance at lower cost would be advantageous.